Regional Educational 
Institute of APPENDIXES Laboratory Mid-Atlantic 


Education Sciences November 2021 


At Mathematica 


Impacts of Home Visits on Students in District of Columbia 
20] 0) (oso afore) fs 


Appendix A. Literature review 
Appendix B. Methods 
Appendix C. Baseline equivalence 


Appendix D. Explanation for dropping the analysis of Family Engagement Collaborative home visits 


See hitps://go.usa.gov/xertp for the full report. 


Appendix A. Literature review 

Structured relationship-building teacher home visits have emerged as a promising method to increase parent 
engagement and improve student outcomes, but there is limited evidence on their effectiveness. Two meta- 
analyses found that family engagement programs, including those focused on parent-teacher relationships, 
collaboration, and communication, positively impacted student academic achievement and social-emotional and 
mental health outcomes. But the analyses were not restricted to studies with rigorous designs, and many of the 
studies had small sample sizes (Jeynes, 2012; Sheridan et al., 2019). Based on interviews and debriefing sessions 
with parents and teachers participating in home visits, another study suggested that home visits have positive 
effects on students’ feelings about their school and their teachers (Goff Pejsa & Associates, 2014). Among the 
small number of more rigorous studies, a matched comparison group study of a home visit program in a Texas 
charter school system found that the students who received home visits had higher math and English language 
arts course grades, higher levels of positive classroom behavior, and parents who logged into the system’s parent 
portal more frequently (Wright et al., 2018). However, a matched comparison study and a related random 
assignment study of a home visit program implemented in the St. Louis, Missouri, area found inconsistent results. 
Some of the matched comparison analyses found positive results on student attendance and reading assessment 
scores, while the random assignment analyses and some of the matched comparison analyses found no 
statistically significant impacts on attendance, reading assessment scores, or disciplinary referrals (Scher, 2016; 
Scher & Lauver, in press). 


Two studies have involved home visits from the Family Engagement Partnership (FEP) or its home visit model. A 
Johns Hopkins University study on the FEP in the District of Columbia found that students whose families received 
home visits had better attendance and were more likely to score above grade-level proficiency targets on a 
reading comprehension test than students whose families did not receive a visit (Sheldon & Jung, 2015). The study 
also found that teachers who received FEP training scored better on some components of the district’s teacher 
observation measure (leading well-organized objective-driven lessons, providing students multiple ways to move 
toward mastery, responding to student understanding, and developing higher-level understanding). The study 
focused on 12 schools and did not use a rigorously matched comparison group. Another Johns Hopkins University 
study examined home visits in four districts. The home visits were based on the Parent Teacher Home Visits model 
that the FEP is also based on. The study found that the home visits were associated with lower rates of chronic 
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absenteeism and higher rates of proficiency on math and English language arts assessments (Sheldon & Jung, 
2018). The study involved a comparison group but did not demonstrate that the students and schools in the 
comparison group were similar to those in the home visit group. 
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Appendix B. Methods 
This appendix discusses the study’s data sources, sample, and methodology. 


Data sources 


The study team obtained data on Family Engagement Partnership (FEP) home visits; student demographics, 
enrollment, achievement, attendance, and disciplinary incidents; and census block group characteristics (table 
B1). The data covered the 2012/13-—2016/17 school years, except the home visit data, which covered the 2014/15-— 
2016/17 school years.* The study focused on students in grades 1-5 during the three years of the study period; 
however, the study also obtained kindergarten data to use as baseline data for matching. The Flamboyan 
Foundation owns the FEP data and provided the data to the study team via District of Columbia Public Schools 
(DCPS). DCPS owns and provided all other data, except for the census block group characteristics, which are 
publicly available on the U.S. Census Bureau website (https://data.census.gov). The study team matched FEP 
home visit data to DCPS administrative data using student name and other information. 


The study also obtained DCPS administrative data on teacher background characteristics and teaching 
assignments. Ultimately, the study found that teachers and students could not be reliably linked, so the study did 
not include teacher characteristics in the analysis. 


Table B1. Study data, by type of variable 
Type Description 


Treatment variable 


Family Engagement Name, grade, and school of students who received home visits, names of teachers who provided 
Partnership home visit those visits, and the dates of the visits. The study team obtained similar data on Family 
data Engagement Collaborative visits, but the analysis of visits under that program was dropped from 


the study (see appendix D). 
Background variables 


Census block group Five-year averages of the percentage of individuals age 25 and older by highest educational 
characteristics attainment for each census block group. The study team attempted to match students to their 
census block group characteristics using student home address data. 


Student demographics Student-level race/ethnicity, gender, age, free or reduced-price lunch status, special education 
status, English learner status, and home address. 


Student enrollment Student-level grade, school, admission date, withdrawal date, membership days (number of days 
enrolled), excused absences, and unexcused absences. 


1 The study team obtained home visit data for the 2013/14-2016/17 school years. However, the FEP data system did not maintain the dates 
of 2013/14 FEP visits. Restricting the sample of visited students to those who received their first FEP visit of a given school year during the 
summer before the start of that school year thus resulted in excluding 2013/14 as a study year. 
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Type Description 


Outcome variables 


Student achievement Student-level math and English language arts z-scores on the District of Columbia Comprehensive 
Assessment System (DC CAS) and Partnership for Assessment of Readiness for College and Careers 
(PARCC) assessment, which were the annual standardized assessments DCPS administered during 
the study period. Scores are from the DC CAS for the 2012/13 and 2013/14 school years and from 
the PARCC assessment for the 2014/15-2016/17 school years. The analysis included test scores 
for students in grades 3 (baseline for grade 4), 4, and 5. To express scores in a common unit, DC 
CAS and PARCC scale scores for the District of Columbia Public Schools (DCPS) test-taking 
population were converted to z-scores using the means and standard deviations for each 
assessment. Specifically, the study team calculated math and English language arts scale score 
means and standard deviations by grade-year among all full school year students who had a valid 
test score and were tested on grade level. Subtracting the mean score of this population from an 
individual student’s scale score and then dividing that difference by the standard deviation of 
scores in the population yields a z-score. Z-scores have a mean of 0 and a standard deviation of 1 
within each grade-year. Impacts on z-scores thus represent changes to achievement expressed in 
standard deviations within the referenced DCPS test-taking population. 


Student attendance rate Calculated from student enrollment data noted above as: 
excused absences + unexcused absences 
( os : ) x 100 
membership days 
Student disciplinary Date and tier of each disciplinary incident. DCPS categorizes incidents into five tiers based on the 
incidents severity of the incident and prescribed consequences: 


Tier 1 incidents (for example, behaviors that disrupt or interfere with classroom teaching and 
learning; running in the classroom, hall, or building; directing profanity or obscene or 
offensive gestures toward peers) result in classroom-level discipline, but teachers can seek a 
disciplinary response from an administrator or a school-level committee if the behavior 
continues. 

Tier 2 incidents (for example, inappropriate or disruptive physical contact between students; 
intentional misuse of school equipment, supplies, or facilities; documented pattern of 
persistent Tier 1 behavior) lead to an administrative or school-based administrative response. 
Tier 3 incidents (for example, engaging in reckless behavior that may cause harm to self or 
others; fighting where there is no injury and no weapon; bullying, or using humiliating or 
intimidating language or behavior) could result in suspension (either onsite or offsite). 

Tier 4 incidents (for example, possession of a weapon or replica or imitation of a weapon, 
fighting that creates substantial risk of or results in minor injury, sexual harassment) could 
lead to offsite suspension. 

Tier 5 incidents (for example, use, threatened use, or transfer of any weapon; assault or 
physical attack on student or staff; possession of drug paraphernalia or controlled substance) 
could result in offsite suspension or expulsion. 


The study team used these data to create three measures for each student-year: 


Source: Authors’ compilation. 


A binary indicator for whether a student had a disciplinary incident (an incident that falls 
under any of the five tiers). 

A binary indicator for whether the student had a serious disciplinary incident (an incident that 
falls under Tier 3, 4, or 5). 

The number of days that a student had a disciplinary incident during a given school year. 


Sample 


The study sample included DCPS students who were in grades 1-5 at any point during the 2014/15—2016/17 
school years. To be included in the sample for a given school year, a student had to be enrolled in a traditional 
DCPS school—that is, a school not designated as an adult education, special education, or youth engagement 
school—and must have remained in the same school from October 1 through June 1 of that school year. The latter 
enrollment criterion helped ensure that students were present in the school long enough to see any initial impacts 
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from the home visits. The study excluded from the eligible sample student-year observations identified as being 
tested off grade, missing current-year demographic variables, or missing valid outcome data. The analysis further 
restricted the sample of visited students to those who received the first FEP visit of a given school year during the 
summer before the start of that school year. Focusing on summer FEP visits ensured that neither visited nor 
comparison students had a disciplinary incident before the first visit of the school year. Additional restrictions to 
the sample resulted from the matching process, described below, which resulted in home visit and comparison 
groups that were similar at baseline (see appendix C). The demographics of FEP-visited students in the analytic 
sample were broadly similar to those of DCPS students as a whole on several characteristics, but there were 
differences of more than 5 percentage points between the two groups in the percentages of students who were 
Black, Hispanic, English learner students, or free lunch recipients (table B2). 


Table B2. Demographic characteristics of Family Engagement Partnership (FEP)—visited students and all 
District of Columbia Public Schools (DCPS) students in grades 1-5 


W/Kex-Tam (eX=1ae=al 9) 


Characteristic All DCPS 
Male 51.56 51.59 
Black 54.79 62.12 
Hispanic 30.39 19.23 
White 11.29 14.32 
Other race/ethnicity 3.53 4.33 
English learner student 18.95 11.69 
Received free lunch 83.97 77.50 
Was a Special education student 16.08 14.15 
Over age for grade 1.66 1.19 
Number of student-year observations 1,984 59,764-62,451 


Note: The FEP group consists of FEP-visited students in the analytic sample, which excludes student-year observations that were missing demographic data. 
Sample sizes for the all DCPS group differ across characteristics because the number of student-year observations missing demographic data varies across 
characteristics. 

Source: Authors’ analysis of District of Columbia Public Schools administrative data and Flamboyan Foundation data on Family Engagement Partnership 
home visits. 


Methodology 


The study team used propensity score matching at the student level to estimate the impact of receiving a home 
visit. Specifically, student data were used to match students who received a home visit (the home visit group) to 
similar students who did not receive a visit (the comparison group), thereby creating visited and nonvisited groups 
that were equivalent on observed preintervention (that is, baseline) measures. The impact of home visits was then 
estimated using regression analysis to compare outcomes for the two groups. The analyses combined all student- 
year observations across the three school years of the study period to estimate an average impact of the home 
visits in the year immediately following a summer home visit. 


Imputation. Before conducting propensity score matching, the study team used multiple imputation to impute 
missing student background characteristics and baseline measures of the outcomes. Missing outcome data were 
not imputed; as noted above, students missing valid outcome data were excluded from the sample. The Stata MI 
suite of commands was used to estimate imputation regression models separately by intervention status (home 
visit or comparison), grade group (grades 1-3 or grades 4 and 5), and year. These imputation models included all 
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the variables used as covariates in the impact estimation models, as well as the outcome variables.” Five sets of 
imputations and the Stata MI commands were then used to estimate impacts. Using multiple sets of imputations 
enabled the study team to adjust standard errors for the variability created by imputing data. Table B3 shows the 
percentage of each baseline characteristic that was imputed for the main analytic sample. 


Table B3. Percentages of baseline characteristic data that were imputed 


Baseline characteristic Grades 1-3 Grades 4and5 
Number of days with disciplinary incidents 5.37 3.09 
Attendance rate 5.45 3.09 
Math achievement na 6.49 
English language arts achievement na 6.86 
Male 0 0 
Black 0 0 
Hispanic 0 0 
White 0 0 
Other race/ethnicity 0 0 
English learner student 4.92 2.34 
Received free lunch 4.35 2.34 
Was a special education student 5.22 2.79 
Over age for grade 3.41 1.66 
Percentage of individuals in the census block group 4.39 3.09 


who did not have a high school diploma or GED 


Number of student-year observations 2,670 1,326 


na is not applicable. 

Note: Percentages were calculated using weights to account for some home visit students being matched to multiple comparison students. 

Source: Authors’ analysis of District of Columbia Public Schools administrative data, U.S. Census Bureau data files, and Flamboyan Foundation data on Family 
Engagement Partnership home visits. 


Propensity score matching. The process for matching individuals and selecting comparison groups consisted of 
several steps. First, the study team estimated grade group—by-year logistic regression models that predicted the 
probability of students receiving a home visit, where the grade group is grades 1-3 or grades 4 and 5. For each of 
the three years of outcome data, the pool of eligible comparison group members was restricted to students in the 
same grade group as the home visit students. The dependent variable of the logistic model of propensity to receive 
a home visit was an indicator coded as 1 for students who received a visit in that year and O for students who did 
not. Each grade group—by-year logistic regression model included the following variables: 


? Specifically, variables in the imputation regression models included student gender, race/ethnicity, grade, and outcomes; baseline 
measures of the student’s outcome, free lunch status, English learner status, over-age for grade status, the percentage of individuals in the 
student’s census block group who did not have a high school diploma or GED, the percentage of individuals in the student’s census block 
group who had a bachelor’s degree or higher, and the percentage of individuals in the student’s census block group who were below 185 
percent of the poverty level; school-grade-year means of gender, race/ethnicity, and the preceding baseline characteristics; and a baseline 
measure of the student’s special education status. The imputation regression models for grades 1-3 did not include student achievement 
outcomes; the models for grades 4 and 5 included student achievement outcomes as right-hand-side variables and baseline measures of 
those student achievement outcomes as left-hand-side variables. 
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At the student level (for student i in grade g in school s) 

e Number of days with disciplinary incidents in the prior year. 

e Prior-year attendance rate. 

e Prior-year math achievement z-score (for grades 4 and 5 only). 

e Prior-year English language arts achievement z-score (for grades 4 and 5 only). 
e Indicator for male. 

e Race/ethnicity indicators. 

e Indicator for being an English learner student in the prior year. 

e Indicator for receiving free lunch in the prior year. 

e Indicator for being a special education student in the prior year. 


e Percentage of individuals age 25 and older in the prior year census block group who did not have a high school 
diploma or GED. 


e Grade indicators. 

At the school-grade level 

e Mean attendance rate for the same cohort (school s, grade g — 1) in the prior year. 

e Mean number of days with disciplinary incidents for the same cohort in the prior year. 

e Mean math achievement z-score for same cohort in the prior year (for grades 4 and 5 only). 

e Mean English language arts achievement z-score for same cohort in the prior year (for grades 4 and 5 only). 


e Mean percentage of individuals age 25 and older in census block group who did not have a high school diploma 
or GED for the same cohort in the prior year. 


Estimating these logistic regression models produced a set of predicted probabilities (propensity scores) for each 
of the five imputations. The study team then calculated the mean propensity score across imputations for each 
grade group—year. 


Next, the study team applied a matching algorithm—nearest neighbor without replacement—to the estimated 
mean propensity scores to create a comparison group for the home visit group. Matching was conducted 
separately for each grade-year, thus ensuring that comparison group students were in the same grade in a given 
outcome year as the home visit students to whom they were matched. For the analysis of FEP, students who 
received an FEP visit after the start of a given school year and students who received an FEC visit at any point 
during that school year were excluded from the potential comparison group. The frame for the comparison group 
thus included all nonvisited students in traditional DCPS schools who met the sample grade and enrollment 
criteria. 


Before reviewing any outcome data, the study team selected nearest neighbor without replacement as the 
matching algorithm. Each home visit student was matched to the nonvisited student with the most similar 
propensity score. A nonvisited student could be a nearest neighbor for only one treatment student. In the event 
of ties (multiple nonvisited nearest neighbors for a given visited student), all nonvisited nearest neighbors were 
included, and the weights for the set of nearest neighbors were normalized to 1. This approach produced home 
visit and comparison groups that were intended to meet What Works Clearinghouse version 4.1 baseline 
equivalence standards for all five outcomes of interest in each of the three outcome years (see appendix C). 
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Impact estimation. The study team estimated the impact of home visits by comparing the outcomes for home visit 
students to the outcomes for the matched students using regressions. 


Impacts on student disciplinary incidents, attendance, and achievement were estimated using the following 
equation: 


Vit = A+ X;B+ ST; + year dummy variables + €;, 


where yi is the outcome for student jin year t; His the intercept term; X;is a vector of baseline student and school- 
grade characteristics (including a baseline measure of the outcome, demographic characteristics, grade indicators, 
and census block group characteristics data for students); Tit is a treatment indicator that equals 1 if the student 
received a home visit in that year and O otherwise; €; is a random error term that reflects the influence of 
unobserved factors on the outcome; and a, B, and 6 are parameters to be estimated. The parameter 6 represents 
the impact of a home visit. 


The study team estimated the models with ordinary least squares using robust standard errors that accounted for 
the clustering of students within school-grade-years. Data for the three outcome years were stacked to create a 
single estimate of the impact of a home visit for the school year immediately following the summer home visit. 
The analytic weights applied accounted for each home visit student potentially having multiple matched 
comparison students. Stata MI commands were used to adjust standard errors for variability within and between 
imputations, as well as the number of imputations (in this case, five). 
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Appendix C. Baseline equivalence 

This appendix provides analyses showing that the home visit and matched comparison groups were similar on key 
measures before teachers visited Family Engagement Partnership students (that is, at baseline). Tables C1 and C2 
present differences in baseline means between the analytic home visit and comparison groups. Table C3 reports 
data in support of the study’s intention to meet What Works Clearinghouse version 4.1 baseline equivalence 
standards for all five outcomes of interest in each of the three outcome years. 


Table C1. Baseline characteristics of analytic samples for impacts of Family Engagement Partnership summer 
visits on disciplinary incidents and student attendance in grades 1-5 
Mean 


Difference Syielalerlae| 


Baseline characteristic Home visit Comparison ameatexclals lagelg 


Student level 


Number of days with disciplinary incidents 0.19 0.22 -0.03 0.05 .574 
Attendance rate 95.15 95.09 0.06 0.22 .770 
Male 51.56 50.30 1.26 1.63 .439 
Black 54.79 56.50 -1.71 3.86 .657 
Hispanic 30.39 31.05 —0.66 3.89 .866 
White 11.29 8.87 2.42 2.20 .272 
Other race/ethnicity 3.53 3.58 —0.05 0.71 943 
English learner student 22.24 22.25 —-0.01 3.35 998 
Received free lunch 83.25 86.39 —-3.15 3.38 352 
Was a Special education student 15.40 15.79 —0.38 1.47 795 
Over age for grade 1.46 0.90 0.56 0.36 .116 
Percentage of individuals in census block group 15.40 15.55 -0.15 0.62 .809 


who do not have a high school diploma or GED 


School-grade level 


Number of days with disciplinary incidents 0.21 0.20 0.01 0.03 797 
Attendance rate 93.72 93.74 —0.02 0.29 .948 
Male 50.46 50.59 —0.13 0.64 .839 
Black 60.36 64.09 -3.73 3.43 .277 
Hispanic 27.02 24.72 2.30 3.39 498 
White 9.00 8.08 0.92 1.79 .610 
Other race/ethnicity 3.62 3.10 0.52 0.42 .222 
English learner students 19.59 19.13 0.46 2.81 869 
Received free lunch 84.98 86.90 -1.92 3.02 525 
Special education students 13.92 12.68 1.24* 0.63 .049 
over age for grade 1.18 1.03 0.15 0.18 412 
Percentage of individuals in census block group 15.45 15.59 0.14 0.54 .796 


who do not have a high school diploma or GED 
Number of student-year observations 1,984 2,012 


* Significant at p < .05. 

Note: Home visit and comparison means are unadjusted. For disciplinary incidents, means and differences in means are in days. For all other characteristics, 
means are in percentages, and differences in means are in percentage points. The estimation of the difference in means used robust standard errors that 
accounted for clustering of students within school-grade-years and a weight to ensure that standard errors accounted for some home visit students being 
matched to multiple comparison students. 

Source: Authors’ analysis of District of Columbia Public Schools administrative data, U.S. Census Bureau data files, and Flamboyan Foundation data on Family 
Engagement Partnership home visits. 
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Table C2. Baseline characteristics of analytic samples for impacts of Family Engagement Partnership summer 
visits on math and English language arts achievement in grades 4 and 5 


M 
al Difference Niclalerlae| 


Baseline characteristic Home visit (olanley-lanxela amanda lanelg 


Student level 


Math achievement —-0.11 —0.07 —0.04 0.08 .674 
English language arts achievement —0.16 —0.18 0.02 0.09 .796 
Male 52.64 50.98 1.66 3.10 593 
Black 63.05 63.20 -0.15 6.52 982 
Hispanic 27.00 28.96 -1.96 6.29 .755 
White 7.69 5.88 1.81 2.78 515 
Other race/ethnicity 2.26 1.96 0.30 0.85 .724 
English learner student 13.82 14.66 —0.84 4.38 .847 
Received free lunch 88.05 89.68 —1.63 4.66 727 
Was a special education student 14.48 14.63 -0.15 2.53 952 
Over age for grade 2.56 1.51 1.06 0.77 173 
Percentage of individuals in census block group 16.07 15.75 0.31 0.83 .706 


who do not have a high school diploma or GED 


School-grade level 


Math achievement —0.17 —0.16 0.00 0.06 .961 
English language arts achievement 0.24 0.23 —0.01 0.06 .889 
Male 51.28 50.26 1.02 1.33 444 
Black 67.63 68.90 —1.27 5.72 .824 
Hispanic 24.82 23.07 1.76 5.53 751 
White 4.81 5.32 —0.50 1.80 779 
Other race/ethnicity 2.74 2.72 0.02 0.52 .970 
English learner students 14.09 15.01 0.93 3.88 .811 
Received free lunch 90.40 90.64 0.24 3.71 948 
Special education students 15.36 13.61 1.74 0.95 .067 
Over age for grade 2.10 1.72 0.39 0.38 312 
Percentage of individuals in census block group 16.41 16.31 0.10 0.74 .890 


who do not have a high school diploma or GED 
Number of student-year observations 663 663 


Note: Home visit and comparison means do not differ by a statistically significant margin at p < .05 for any characteristic. Achievement refers to scores on 
the District of Columbia Comprehensive Assessment System (DC CAS) for the 2013/14 school year and scores on the Partnership for Assessment of Readiness 
for College and Careers (PARCC) assessment for the 2014/15—2016/17 school years. To express scores in a common unit, DC CAS and PARCC scale scores for 
the DCPS test-taking population were converted to z-scores using the means and standard deviations for each assessment. Z-scores have a mean of O anda 
standard deviation of 1 within each grade-year. Home visit and comparison means are unadjusted. For math and English language arts achievement, means 
and differences in means are in z-scores. For all other characteristics, means are in percentages, and differences in means are in percentage points. The 
estimation of the difference in means used robust standard errors that accounted for clustering of students within school-grade-years and a weight to ensure 
that standard errors accounted for some home visit students being matched to multiple comparison students. 

Source: Authors’ analysis of District of Columbia Public Schools administrative data, U.S. Census Bureau data files, and Flamboyan Foundation data on Family 
Engagement Partnership home visits. 
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Table C3. Characteristics used to assess What Works Clearinghouse baseline equivalence requirements for analysis of the impacts of Family Engagement 


Partnership summer visits 


Correlation 
Sample size Outcome Baseline Sample size Outcome Baseline between 
baseline 
Full sample Observed Full sample Observed late) 
Observed baseline Observed baseline outcome 
Observed baseline Full sample Observed baseline Full sample (observed 
Full baseline Standard sample sample standard Full baseline Standard sample sample standard baseline 
Outcome variable sample sample deviation mean mean deviation sample Tan} e) 2) Mean deviation fater-la} mean deviation sample) 
2014/15 
Had a disciplinary incident 778 757 0.095 0.294 0.094 0.188 1.127 788 735 0.129 0.336 0.126 0.245 1.533 0.259 
Had a serious disciplinary 778 757 0.089 0.284 0.087 0.188 1.127 788 735 0.101 0.301 0.100 0.245 1.533 0.268 
incident 
Attendance rate 778 757 0.950 0.057 0.949 0.953 0.048 788 735 0.951 0.047 0.950 0.952 0.044 0.619 
ELA achievement 202 193 0.093 0.885 —0.062 —0.050 0.910 202 184 —0.180 0.839 —0.183 —0.145 0.894 0.797 
Math achievement 202 193 0.125 0.948 0.154 0.033 0.882 202 185 0.076 0.827 0.077 0.018 0.823 0.792 
2015/16 
Had a disciplinary incident 843 816 0.071 0.257 0.071 0.187 1.166 857 792 0.101 0.301 0.103 0.201 1.559 0.242 
Had a serious disciplinary 843 816 0.065 0.247 0.066 0.187 1.166 857 792 0.077 0.267 0.078 0.201 1.559 0.264 
incident 
Attendance rate 843 816 0.959 0.038 0.959 0.951 0.044 857 790 0.951 0.045 0.951 0.948 0.046 0.615 
ELA achievement 294 275 —0.109 0.889 -0.079 -0.179 0.888 294 260 —0.180 0.918 —0.168 —0.212 0.928 0.833 
Math achievement 294 278 —0.032 0.952 0.014 —0.137 0.904 294 261 -0.075 0.902 —0.050 0.078 0.901 0.853 
2016/17 
Had a disciplinary incident 363 354 0.138 0.345 0.141 0.201 0.804 367 359 0.157 0.364 0.158 0.193 0.929 0.332 
Had a serious disciplinary 363 354 0.094 0.292 0.096 0.201 0.804 367 359 0.096 0.296 0.096 0.193 0.929 0.355 
incident 
Attendance rate 363 354 0.949 0.044 0.949 0.951 0.045 367 359 0.943 0.054 0.943 0.954 0.039 0.611 
ELA achievement 167 165 —0.266 0.930 0.265 —0.256 0.910 167 158 0.161 0.811 0.142 0.173 0.839 0.833 
Math achievement 167 165 —0.100 0.865 0.095 0.228 0.894 167 158 —0.105 0.916 0.053 -0.175 0.946 0.854 


ELA is English language arts. 


Home visit group 


Comparison group 


Note: Achievement refers to scores on the District of Columbia Comprehensive Assessment System (DC CAS) for the 2013/14 school year and scores on the Partnership for Assessment of Readiness for College and 
Careers (PARCC) assessment for the 2014/15—2016/17 school years. To express scores ina common unit, DC CAS and PARCC scale scores for the DCPS test-taking population were converted to z-scores using the means 
and standard deviations for each assessment. Z-scores have a mean of 0 and a standard deviation of 1 within each grade-year. 

Source: Authors’ analysis of District of Columbia Public Schools administrative data and Flamboyan Foundation data on Family Engagement Partnership home visits. 


Appendix D. Explanation for dropping the analysis of Family Engagement Collaborative home 
visits 

The study originally attempted to include structured, relationship-building home visits from a second DCPS 
program—the Family Engagement Collaborative (FEC)—in analyzing the impacts of home visits on student 
disciplinary incidents, attendance, and achievement (research questions 1-3) and to answer two additional 
secondary research questions: 


4. Dothe impacts of the home visits differ by home visit program? 


5. Do home visits conducted earlier in the school year impact student disciplinary incidents and attendance 
differently than home visits conducted later in the school year? 


Although the Family Engagement Partnership (FEP) and the FEC have important differences, including program 
components beyond home visits and whether the programs are schoolwide (FEP) or teacher level (FEC), the 
programs’ home visit training and procedures are identical. DCPS expressed interest in the impacts of the visits 
regardless of program context, and evaluating the impacts of the home visits from both programs together would 
have increased the statistical power of the analysis. Questions 4 and 5 focused on potentially actionable aspects 
of the programs for which DCPS did not have clear expectations. The similarity between the home visit 
components of the FEP and FEC made it unclear whether to expect different impacts by program (question 4). 
Regarding visit timing, on the one hand, an early start at relationship-building might result in earlier visits having 
a greater impact than later ones; on the other hand, conducting visits later in the school year might enable 
teachers to leverage their experiences to date with their students and use the visits strategically to better 
encourage positive behaviors and discourage negative behaviors (question 5). 


The study team excluded FEC visits from the analysis after finding evidence suggesting that teachers might have 
selected students to visit in a given school year based, at least in part, on their observations of students during 
that school year. Such selection would have biased estimated impacts of home visits conducted during the school 
year, even if the groups of visited and comparison students were equivalent on measured characteristics at 
baseline (before the start of the school year). Restricting the impact analysis to summer visits eliminated this 
particular mechanism for selection bias. However, this restriction could not be applied to FEC visits because of the 
relatively small FEC sample size (table D1); as a result, the study team dropped question 4 and examined questions 
1-3 only for the (larger) FEP program. The selection bias evidently associated with visits conducted during the 
school year meant that the study could not produce a credible analysis of the impact of visit timing, leading to 
question 5 being dropped as well. 


Table D1. Roughly 10 percent of visited students in the eligible study sample received a Family Engagement 
Collaborative home visit, and 90 percent received a Family Engagement Partnership visit 


Visited students, by school year 


2014/15 2015/16 2016/17 2014/15-2016/17 
Home visit program Percent Number Percent Number Percent Number’ Percent Number 
Family Engagement Collaborative? 14.87 466 9.03 318 6.12 310 9.33 1,094 
Family Engagement Partnership 85.13 2,668 90.97 3,203 93.88 4,757 90.67 10,628 


a. Includes visits by Family Engagement Collaborative Masters teachers. 
Source: Authors’ analysis of District of Columbia Public Schools administrative data and data on Family Engagement Collaborative home visits and Flamboyan 
Foundation data on Family Engagement Partnership. 


After describing the FEC program, the remainder of this appendix provides additional details about the decision 
to drop FEC visits from the analysis. 
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Description of the Family Engagement Collaborative program 

Developed by DCPS in partnership with the Flamboyan Foundation, the FEC is a teacher-level program available 
to pairs of preschool-grade 5 teachers not in FEP schools. The FEC involves the same home visit training and 
procedures as the FEP but does not include any of the FEP’s other components (table D2). The FEC also involves 
professional learning communities for participating teachers. Teachers must apply to participate in the FEC in pairs 
from the same school. DCPS accepts almost all applicants; however, a small number of applicants are rejected due 
to capacity constraints or an inability to find a partner teacher with whom to participate. In addition to the home 
visit training, FEC teachers can participate in six, two-hour professional learning community meetings facilitated 
by former FEC teachers that cover the following topics: starting off the year right; ongoing communication; goal 
setting; the role of race, class, and culture in family engagement; family engagement innovation; and reflecting 
on the year. The FEC also has a yearlong FEC Masters fellowship program for teachers who complete the initial 
FEC program. The FEC Masters program involves home visits, additional professional learning community 
meetings, and a school family engagement project. FEC and FEC Masters teachers who attend at least five of the 
six professional learning community meetings and conduct 10-24 home visits during the program year receive 
$1,500. FEC and FEC Masters teachers are allowed to visit the same student’s family more than once, but repeat 
visits are rare (less than 1 percent of all visits). As a teacher-level program the FEC is much smaller than the FEP, 
conducting about 500 home visits per year compared with about 11,000 home visits conducted through the FEP 
annually (Hagan, personal communication, May 25, 2021). 
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Table D2. Family Engagement Collaborative (FEC) program components 


Component 
Program level 


Home visit 
training provider 


Home visit 
training content 


Home visit 
procedure 


Additional 
program 
components and 
training during 
the school year 


Compensation 


Details 
Pairs of teachers (FEC Masters also allows groups of three or more) 


District of Columbia Public School and Flamboyan Foundation staff 


e Based on Parent Teacher Home Visits model 
e Lasts two to three hours 
e Takes place over the summer 
e Involves: 
© Parents and teachers sharing home visit experiences 
© Key components of home visits 
© How to approach, plan, schedule, and conduct visits 
o How to sustain families’ trust and maintain communication after visits 
© How to overcome common challenges 


e Home visits are voluntary for both teachers and families. 
e Teachers select who to visit and when. 
e General expectation is one visit per family. 
e Teachers conduct visits in pairs. 
e Visits typically last 30 minutes. 
e Visits take place over the summer or during the school year. 
e Visits involve: 
© Introductions and getting to know each other. 
© Parents’ hopes and dreams for their child’s future. 
© Parents’ expectations for their child’s education. 
© An invitation to continue building a relationship. 
© Parents’ communication preferences. 


e Six, two-hour professional learning community meetings on the following topics: 
© Starting off the year right 

Ongoing communication 

Goal setting 

The role of race, class, and culture in family engagement 

Family engagement innovation 


oO 0 90 0 0 


Reflecting on the year 
e Teachers receive $1,500 if they attend at least five professional learning community meetings and 
conduct the required number of home visits per year: 
o 24 visits for a pair of FEC teachers. 
o 10 visits for a pair of FEC Masters teachers. 
o 18 visits for a group of three or more FEC Masters teachers. 
e Repeat visits to same student’s family count toward the visits. 


Source: Based on training materials provided by and conversations with District of Columbia Public School staff. 


Reasons for dropping the Family Engagement Collaborative program from the analysis 


In attempting to analyze the impacts of both FEP and FEC home visits, the study team created an FEP-visited group, 
an FEC-visited group, and a matched comparison group for each program using the same methods described in 
appendix B with two exceptions: all visited students who met the sample criteria, regardless of visit timing, were 
included (rather than restricting the sample to students who received summer visits), and data from four outcome 
years were used (2013/14—2016/17, rather than excluding the first outcome year because FEP visit dates for 
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2013/14 were unavailable). After estimating the grade group—by-year logistic regression model described in 
appendix B separately for the FEP and the FEC, the study team used the nearest neighbor matching without 
replacement algorithm to produce a comparison group for each program. The resulting home visit and comparison 
groups were intended to meet What Works Clearinghouse version 4.1 baseline equivalence standards for all five 
outcomes of interest—had a disciplinary incident, had a serious disciplinary incident, attendance rate, math 
achievement, and English language arts achievement—in all four outcome years for the FEP analysis and in all 
outcome years except 2013/14 for the FEC analysis.2 


However, comparing the percentages of home visit and comparison students who had disciplinary incidents 
before the first visit of the school year yielded evidence that undermined the credibility of an analysis of visits 
conducted during the school year. For this comparison the study team identified the first home visit date of the 
school year for each visited student and examined whether visited students and their matched comparison 
students had a disciplinary incident before that visit date. More than 13 percent of students visited by FEC teachers 
had a disciplinary incident before the visit compared with nearly 9 percent of matched comparison students, a 
statistically significant difference of more than 4 percentage points (table D3). Any estimated impact of the FEC 
visits could be biased by students who had disciplinary incidents being more likely to receive a visit than students 
who did not have disciplinary incidents. The more modest yet also statistically significant difference found for FEP 
suggested selection bias in the opposite direction—that is, FEP students were about 1 percentage point less likely 
than their matched comparison students to have had a disciplinary incident by the time of the visit. 


Table D3. Relative to their matched comparison students, Family Engagement Collaborative students were 
more likely and Family Engagement Partnership students were less likely to have had a disciplinary incident 
by the time of the visited student’s first visit of the school year 

Mean (percent) Difference 


(percentage Standard 
Outcome Home visit Comparison reXe) Tay ks)) lacey e p-value 


Family Engagement Collaborative (FEC)? 


Had a disciplinary incident before the FEC student’s visit 13.09 8.55 4.54*** 1.27 < .001 
Number of student-year observations 1,094 1,101 

Family Engagement Partnership (FEP) 

Had a disciplinary incident before the FEP student’s visit 2.56 3.71 —1.16*** 0.33 < .001 
Number of student-year observations 10,628 10,732 


*** Significant at p < .001. 

Note: The comparison group mean is the unadjusted mean outcome for the comparison group. The home visit mean is the regression-adjusted mean 
outcome for the home visit group and is equal to the comparison group mean plus the difference estimate. The regressions used to estimate differences 
controlled for student gender, race/ethnicity, and grade; baseline measures of the student’s outcome, free lunch status, English learner status, over-age for 
grade status, and the percentage of individuals in the student’s census block group who did not have a high school diploma or GED; school-grade-year means 
of gender, race/ethnicity, and the preceding baseline characteristics; a baseline measure of the student’s special education status; and year. The regressions 
used robust standard errors that accounted for clustering of students within school-grade-years and a weight to ensure that standard errors accounted for 
some home visit students being matched to more than one comparison student. 

a. Includes visits by Family Engagement Collaborative Masters teachers. 

Source: Authors’ analysis of District of Columbia Public Schools administrative data and data on Family Engagement Collaborative home visits and Flamboyan 
Foundation data on Family Engagement Partnership and . 


1 As noted in appendix B, propensity score matching for the FEP analysis excluded from the potential comparison group for a given school 
year students who received FEC visits at any point during that school year. The FEC analysis implemented the analogous exclusion—that is, 
matching for the FEC analysis dropped FEP-visited students from the comparison group. 

? For 2013/14 the study team’s calculations indicated that the FEC visit and comparison groups met the What Works Clearinghouse baseline 
equivalence standard for had a disciplinary incident, attendance rate, and English language arts achievement but not for had a serious 
disciplinary incident or math achievement. The study team ordinarily would have attempted to adjust the propensity score matching model 
or algorithm to achieve baseline equivalence for all five outcomes in all four outcome years but did not do so in this case after finding 
evidence of selection bias when analyzing only outcome years that met the baseline equivalence standards. 
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For the FEP analysis the study team addressed this potential selection bias by restricting the sample to students 
who were visited before the start of the school year and their matched comparison students. Teachers could not 
have selected students for these summer visits based on student disciplinary incidents (or any other experiences 
with students) that happened during the subsequent school year. Across the three outcome years for which FEP 
visit dates are available, 1,984 FEP student-year observations, nearly 20 percent of the eligible FEP sample, 
received a summer visit (table D4). 


Table D4. Nearly 20 percent of Family Engagement Partnership students in the eligible study sample received 
their first visit during the summer before the school year started 


Visited students, by school year 


2014/15 2015/16 2016/17 2014/15-2016/17 


Timing of first visit of school year Percent Number Percent Number Percent Number Percent Number 
Before start of the school year 29.16 778 26.32 843 7.63 363 18.67 1,984 
During the school year 70.84 1,890 73.68 2,360 92.37 4,394 81.33 8,644 


Source: Authors’ analysis of District of Columbia Public Schools administrative data and Flamboyan Foundation data on Family Engagement Partnership 
home visits. 


This summer visit approach, however, could not be used for the smaller FEC program. Across all four outcome 
years, only 85 FEC student-year observations (7 percent of the eligible FEC sample) received a visit before the start 
of the school year, rendering the sample too small for meaningful analysis (table D5). 


Table D5. Roughly 7 percent of Family Engagement Collaborative students in the eligible study sample 
received their first visit during the summer before the school year started 


Visited students, by school year 


2013/14- 
2013/14 2014/15 PAO Sy Ks) 2016/17 2016/17 


Timing of first visit of school year Percent Number Percent Number Percent Number Percent Number Percent Number 
Before start of the school year 12.92 27 7.73 36 6.92 22 0.00 0 6.52 85 
During the school year 87.08 182 92.27 430 93.08 296 100.00 310 93.48 1,218 


Note: Includes visits by Family Engagement Collaborative Masters teachers. 
Source: Authors’ analysis of District of Columbia Public Schools administrative data and data on Family Engagement Collaborative home visits. 


